Why might we not want to include all the variables available to us?
Recall: 62 unique variables describing Cannabis strains
New Response variable: Rating
Let’s try every possible subset of variables and pick the best one.
What do we mean by best?
Penalized metrics:
Where does cross-validation come in???
The problem:
Rating ~ Creative
Rating ~ Creative + Energetic
Rating ~ Creative + Tingly
Rating ~ Creative + Energetic + Tingly
\(\vdots\)
62 variables = \(2^{62}\) models = 4.6 quintillion models
Plus cross-validation????
If you have only a few variables, go for it.
In realistic settings, it’s not practical.
We’re going to try it anyway, using the leaps package.
leapsBest model of each size, based on R-squared:
library(leaps)
models <- regsubsets(Rating ~ Creative + Energetic + Tingly,
data = cann, method = "exhaustive")
summary(models)Subset selection object
Call: regsubsets.formula(Rating ~ Creative + Energetic + Tingly, data = cann,
method = "exhaustive")
3 Variables (and intercept)
Forced in Forced out
Creative FALSE FALSE
Energetic FALSE FALSE
Tingly FALSE FALSE
1 subsets of each size up to 3
Selection Algorithm: exhaustive
Creative Energetic Tingly
1 ( 1 ) "*" " " " "
2 ( 1 ) "*" "*" " "
3 ( 1 ) "*" "*" "*"
Why can we automatically determine the best model within each size but we need to do another step to choose the best model across sizes?
leapsNow compare models:
Adj-R-Squared CP BIC
Creative 0.01006460 16.694102 -8.840652
Creative, Energetic 0.01400814 8.454895 -11.303084
Creative, Energetic, Tingly 0.01633805 4.000000 -10.016876
What model would we choose?
Start with all candidate variables in the model.
Drop the worst variable. (p-values or R-squared)
Check if dropping it helped. (penalized metric or cross-validation)
Stop when dropping is no longer good.
leapsReordering variables and trying again:
Subset selection object
Call: regsubsets.formula(Rating ~ ., data = cann, method = "backward",
nvmax = 62)
62 Variables (and intercept)
Forced in Forced out
Creative FALSE FALSE
Energetic FALSE FALSE
Tingly FALSE FALSE
Euphoric FALSE FALSE
Relaxed FALSE FALSE
Aroused FALSE FALSE
Happy FALSE FALSE
Uplifted FALSE FALSE
Hungry FALSE FALSE
Talkative FALSE FALSE
Giggly FALSE FALSE
Focused FALSE FALSE
Sleepy FALSE FALSE
Earthy FALSE FALSE
Sweet FALSE FALSE
Citrus FALSE FALSE
Flowery FALSE FALSE
Violet FALSE FALSE
Diesel FALSE FALSE
`Spicy/Herbal` FALSE FALSE
Sage FALSE FALSE
Woody FALSE FALSE
Apricot FALSE FALSE
Grapefruit FALSE FALSE
Orange FALSE FALSE
Pungent FALSE FALSE
Grape FALSE FALSE
Pine FALSE FALSE
Skunk FALSE FALSE
Berry FALSE FALSE
Pepper FALSE FALSE
Menthol FALSE FALSE
Blue FALSE FALSE
Cheese FALSE FALSE
Chemical FALSE FALSE
Mango FALSE FALSE
Lemon FALSE FALSE
Peach FALSE FALSE
Vanilla FALSE FALSE
Nutty FALSE FALSE
Chestnut FALSE FALSE
Tea FALSE FALSE
Tobacco FALSE FALSE
Tropical FALSE FALSE
Strawberry FALSE FALSE
Blueberry FALSE FALSE
Mint FALSE FALSE
Apple FALSE FALSE
Honey FALSE FALSE
Lavender FALSE FALSE
Lime FALSE FALSE
Coffee FALSE FALSE
Ammonia FALSE FALSE
Minty FALSE FALSE
Tree FALSE FALSE
Butter FALSE FALSE
Pineapple FALSE FALSE
Tar FALSE FALSE
Rose FALSE FALSE
Plum FALSE FALSE
Pear FALSE FALSE
Fruit FALSE FALSE
1 subsets of each size up to 61
Selection Algorithm: backward
Creative Energetic Tingly Euphoric Relaxed Aroused Happy Uplifted
1 ( 1 ) " " " " " " " " " " " " "*" " "
2 ( 1 ) " " " " " " " " "*" " " "*" " "
3 ( 1 ) " " "*" " " " " "*" " " "*" " "
4 ( 1 ) " " "*" " " "*" "*" " " "*" " "
5 ( 1 ) "*" "*" " " "*" "*" " " "*" " "
6 ( 1 ) "*" "*" " " "*" "*" " " "*" "*"
7 ( 1 ) "*" "*" " " "*" "*" " " "*" "*"
8 ( 1 ) "*" "*" " " "*" "*" " " "*" "*"
9 ( 1 ) "*" "*" " " "*" "*" " " "*" "*"
10 ( 1 ) "*" "*" "*" "*" "*" " " "*" "*"
11 ( 1 ) "*" "*" "*" "*" "*" " " "*" "*"
12 ( 1 ) "*" "*" "*" "*" "*" "*" "*" "*"
13 ( 1 ) "*" "*" "*" "*" "*" "*" "*" "*"
14 ( 1 ) "*" "*" "*" "*" "*" "*" "*" "*"
15 ( 1 ) "*" "*" "*" "*" "*" "*" "*" "*"
16 ( 1 ) "*" "*" "*" "*" "*" "*" "*" "*"
17 ( 1 ) "*" "*" "*" "*" "*" "*" "*" "*"
18 ( 1 ) "*" "*" "*" "*" "*" "*" "*" "*"
19 ( 1 ) "*" "*" "*" "*" "*" "*" "*" "*"
20 ( 1 ) "*" "*" "*" "*" "*" "*" "*" "*"
21 ( 1 ) "*" "*" "*" "*" "*" "*" "*" "*"
22 ( 1 ) "*" "*" "*" "*" "*" "*" "*" "*"
23 ( 1 ) "*" "*" "*" "*" "*" "*" "*" "*"
24 ( 1 ) "*" "*" "*" "*" "*" "*" "*" "*"
25 ( 1 ) "*" "*" "*" "*" "*" "*" "*" "*"
26 ( 1 ) "*" "*" "*" "*" "*" "*" "*" "*"
27 ( 1 ) "*" "*" "*" "*" "*" "*" "*" "*"
28 ( 1 ) "*" "*" "*" "*" "*" "*" "*" "*"
29 ( 1 ) "*" "*" "*" "*" "*" "*" "*" "*"
30 ( 1 ) "*" "*" "*" "*" "*" "*" "*" "*"
31 ( 1 ) "*" "*" "*" "*" "*" "*" "*" "*"
32 ( 1 ) "*" "*" "*" "*" "*" "*" "*" "*"
33 ( 1 ) "*" "*" "*" "*" "*" "*" "*" "*"
34 ( 1 ) "*" "*" "*" "*" "*" "*" "*" "*"
35 ( 1 ) "*" "*" "*" "*" "*" "*" "*" "*"
36 ( 1 ) "*" "*" "*" "*" "*" "*" "*" "*"
37 ( 1 ) "*" "*" "*" "*" "*" "*" "*" "*"
38 ( 1 ) "*" "*" "*" "*" "*" "*" "*" "*"
39 ( 1 ) "*" "*" "*" "*" "*" "*" "*" "*"
40 ( 1 ) "*" "*" "*" "*" "*" "*" "*" "*"
41 ( 1 ) "*" "*" "*" "*" "*" "*" "*" "*"
42 ( 1 ) "*" "*" "*" "*" "*" "*" "*" "*"
43 ( 1 ) "*" "*" "*" "*" "*" "*" "*" "*"
44 ( 1 ) "*" "*" "*" "*" "*" "*" "*" "*"
45 ( 1 ) "*" "*" "*" "*" "*" "*" "*" "*"
46 ( 1 ) "*" "*" "*" "*" "*" "*" "*" "*"
47 ( 1 ) "*" "*" "*" "*" "*" "*" "*" "*"
48 ( 1 ) "*" "*" "*" "*" "*" "*" "*" "*"
49 ( 1 ) "*" "*" "*" "*" "*" "*" "*" "*"
50 ( 1 ) "*" "*" "*" "*" "*" "*" "*" "*"
51 ( 1 ) "*" "*" "*" "*" "*" "*" "*" "*"
52 ( 1 ) "*" "*" "*" "*" "*" "*" "*" "*"
53 ( 1 ) "*" "*" "*" "*" "*" "*" "*" "*"
54 ( 1 ) "*" "*" "*" "*" "*" "*" "*" "*"
55 ( 1 ) "*" "*" "*" "*" "*" "*" "*" "*"
56 ( 1 ) "*" "*" "*" "*" "*" "*" "*" "*"
57 ( 1 ) "*" "*" "*" "*" "*" "*" "*" "*"
58 ( 1 ) "*" "*" "*" "*" "*" "*" "*" "*"
59 ( 1 ) "*" "*" "*" "*" "*" "*" "*" "*"
60 ( 1 ) "*" "*" "*" "*" "*" "*" "*" "*"
61 ( 1 ) "*" "*" "*" "*" "*" "*" "*" "*"
Hungry Talkative Giggly Focused Sleepy Earthy Sweet Citrus Flowery
1 ( 1 ) " " " " " " " " " " " " " " " " " "
2 ( 1 ) " " " " " " " " " " " " " " " " " "
3 ( 1 ) " " " " " " " " " " " " " " " " " "
4 ( 1 ) " " " " " " " " " " " " " " " " " "
5 ( 1 ) " " " " " " " " " " " " " " " " " "
6 ( 1 ) " " " " " " " " " " " " " " " " " "
7 ( 1 ) " " " " " " " " "*" " " " " " " " "
8 ( 1 ) " " "*" " " " " "*" " " " " " " " "
9 ( 1 ) " " "*" " " "*" "*" " " " " " " " "
10 ( 1 ) " " "*" " " "*" "*" " " " " " " " "
11 ( 1 ) "*" "*" " " "*" "*" " " " " " " " "
12 ( 1 ) "*" "*" " " "*" "*" " " " " " " " "
13 ( 1 ) "*" "*" "*" "*" "*" " " " " " " " "
14 ( 1 ) "*" "*" "*" "*" "*" " " " " " " " "
15 ( 1 ) "*" "*" "*" "*" "*" " " " " " " " "
16 ( 1 ) "*" "*" "*" "*" "*" " " "*" " " " "
17 ( 1 ) "*" "*" "*" "*" "*" " " "*" " " " "
18 ( 1 ) "*" "*" "*" "*" "*" " " "*" " " "*"
19 ( 1 ) "*" "*" "*" "*" "*" " " "*" " " "*"
20 ( 1 ) "*" "*" "*" "*" "*" " " "*" " " "*"
21 ( 1 ) "*" "*" "*" "*" "*" " " "*" " " "*"
22 ( 1 ) "*" "*" "*" "*" "*" " " "*" "*" "*"
23 ( 1 ) "*" "*" "*" "*" "*" " " "*" "*" "*"
24 ( 1 ) "*" "*" "*" "*" "*" " " "*" "*" "*"
25 ( 1 ) "*" "*" "*" "*" "*" " " "*" "*" "*"
26 ( 1 ) "*" "*" "*" "*" "*" " " "*" "*" "*"
27 ( 1 ) "*" "*" "*" "*" "*" " " "*" "*" "*"
28 ( 1 ) "*" "*" "*" "*" "*" " " "*" "*" "*"
29 ( 1 ) "*" "*" "*" "*" "*" " " "*" "*" "*"
30 ( 1 ) "*" "*" "*" "*" "*" "*" "*" "*" "*"
31 ( 1 ) "*" "*" "*" "*" "*" "*" "*" "*" "*"
32 ( 1 ) "*" "*" "*" "*" "*" "*" "*" "*" "*"
33 ( 1 ) "*" "*" "*" "*" "*" "*" "*" "*" "*"
34 ( 1 ) "*" "*" "*" "*" "*" "*" "*" "*" "*"
35 ( 1 ) "*" "*" "*" "*" "*" "*" "*" "*" "*"
36 ( 1 ) "*" "*" "*" "*" "*" "*" "*" "*" "*"
37 ( 1 ) "*" "*" "*" "*" "*" "*" "*" "*" "*"
38 ( 1 ) "*" "*" "*" "*" "*" "*" "*" "*" "*"
39 ( 1 ) "*" "*" "*" "*" "*" "*" "*" "*" "*"
40 ( 1 ) "*" "*" "*" "*" "*" "*" "*" "*" "*"
41 ( 1 ) "*" "*" "*" "*" "*" "*" "*" "*" "*"
42 ( 1 ) "*" "*" "*" "*" "*" "*" "*" "*" "*"
43 ( 1 ) "*" "*" "*" "*" "*" "*" "*" "*" "*"
44 ( 1 ) "*" "*" "*" "*" "*" "*" "*" "*" "*"
45 ( 1 ) "*" "*" "*" "*" "*" "*" "*" "*" "*"
46 ( 1 ) "*" "*" "*" "*" "*" "*" "*" "*" "*"
47 ( 1 ) "*" "*" "*" "*" "*" "*" "*" "*" "*"
48 ( 1 ) "*" "*" "*" "*" "*" "*" "*" "*" "*"
49 ( 1 ) "*" "*" "*" "*" "*" "*" "*" "*" "*"
50 ( 1 ) "*" "*" "*" "*" "*" "*" "*" "*" "*"
51 ( 1 ) "*" "*" "*" "*" "*" "*" "*" "*" "*"
52 ( 1 ) "*" "*" "*" "*" "*" "*" "*" "*" "*"
53 ( 1 ) "*" "*" "*" "*" "*" "*" "*" "*" "*"
54 ( 1 ) "*" "*" "*" "*" "*" "*" "*" "*" "*"
55 ( 1 ) "*" "*" "*" "*" "*" "*" "*" "*" "*"
56 ( 1 ) "*" "*" "*" "*" "*" "*" "*" "*" "*"
57 ( 1 ) "*" "*" "*" "*" "*" "*" "*" "*" "*"
58 ( 1 ) "*" "*" "*" "*" "*" "*" "*" "*" "*"
59 ( 1 ) "*" "*" "*" "*" "*" "*" "*" "*" "*"
60 ( 1 ) "*" "*" "*" "*" "*" "*" "*" "*" "*"
61 ( 1 ) "*" "*" "*" "*" "*" "*" "*" "*" "*"
Violet Diesel `Spicy/Herbal` Sage Woody Apricot Grapefruit Orange
1 ( 1 ) " " " " " " " " " " " " " " " "
2 ( 1 ) " " " " " " " " " " " " " " " "
3 ( 1 ) " " " " " " " " " " " " " " " "
4 ( 1 ) " " " " " " " " " " " " " " " "
5 ( 1 ) " " " " " " " " " " " " " " " "
6 ( 1 ) " " " " " " " " " " " " " " " "
7 ( 1 ) " " " " " " " " " " " " " " " "
8 ( 1 ) " " " " " " " " " " " " " " " "
9 ( 1 ) " " " " " " " " " " " " " " " "
10 ( 1 ) " " " " " " " " " " " " " " " "
11 ( 1 ) " " " " " " " " " " " " " " " "
12 ( 1 ) " " " " " " " " " " " " " " " "
13 ( 1 ) " " " " " " " " " " " " " " " "
14 ( 1 ) " " " " " " " " " " " " " " " "
15 ( 1 ) " " " " " " " " " " " " " " " "
16 ( 1 ) " " " " " " " " " " " " " " " "
17 ( 1 ) " " " " "*" " " " " " " " " " "
18 ( 1 ) " " " " "*" " " " " " " " " " "
19 ( 1 ) " " " " "*" " " " " " " " " " "
20 ( 1 ) " " " " "*" " " " " " " " " " "
21 ( 1 ) " " " " "*" " " " " " " " " " "
22 ( 1 ) " " " " "*" " " " " " " " " " "
23 ( 1 ) " " " " "*" " " " " " " " " " "
24 ( 1 ) " " " " "*" " " " " " " " " " "
25 ( 1 ) " " " " "*" " " " " " " " " " "
26 ( 1 ) " " "*" "*" " " " " " " " " " "
27 ( 1 ) " " "*" "*" " " " " " " " " " "
28 ( 1 ) " " "*" "*" " " " " " " " " " "
29 ( 1 ) " " "*" "*" " " " " " " " " " "
30 ( 1 ) " " "*" "*" " " " " " " " " " "
31 ( 1 ) " " "*" "*" " " " " " " " " " "
32 ( 1 ) " " "*" "*" " " " " " " " " " "
33 ( 1 ) " " "*" "*" " " " " " " " " " "
34 ( 1 ) " " "*" "*" " " " " " " " " " "
35 ( 1 ) " " "*" "*" " " " " " " " " "*"
36 ( 1 ) " " "*" "*" " " " " " " " " "*"
37 ( 1 ) " " "*" "*" "*" " " " " " " "*"
38 ( 1 ) " " "*" "*" "*" " " " " " " "*"
39 ( 1 ) " " "*" "*" "*" " " " " " " "*"
40 ( 1 ) " " "*" "*" "*" " " " " " " "*"
41 ( 1 ) " " "*" "*" "*" "*" " " " " "*"
42 ( 1 ) " " "*" "*" "*" "*" " " " " "*"
43 ( 1 ) " " "*" "*" "*" "*" " " " " "*"
44 ( 1 ) " " "*" "*" "*" "*" " " " " "*"
45 ( 1 ) " " "*" "*" "*" "*" " " " " "*"
46 ( 1 ) " " "*" "*" "*" "*" " " " " "*"
47 ( 1 ) " " "*" "*" "*" "*" " " " " "*"
48 ( 1 ) "*" "*" "*" "*" "*" " " " " "*"
49 ( 1 ) "*" "*" "*" "*" "*" " " " " "*"
50 ( 1 ) "*" "*" "*" "*" "*" " " " " "*"
51 ( 1 ) "*" "*" "*" "*" "*" " " " " "*"
52 ( 1 ) "*" "*" "*" "*" "*" " " " " "*"
53 ( 1 ) "*" "*" "*" "*" "*" " " " " "*"
54 ( 1 ) "*" "*" "*" "*" "*" "*" " " "*"
55 ( 1 ) "*" "*" "*" "*" "*" "*" "*" "*"
56 ( 1 ) "*" "*" "*" "*" "*" "*" "*" "*"
57 ( 1 ) "*" "*" "*" "*" "*" "*" "*" "*"
58 ( 1 ) "*" "*" "*" "*" "*" "*" "*" "*"
59 ( 1 ) "*" "*" "*" "*" "*" "*" "*" "*"
60 ( 1 ) "*" "*" "*" "*" "*" "*" "*" "*"
61 ( 1 ) "*" "*" "*" "*" "*" "*" "*" "*"
Pungent Grape Pine Skunk Berry Pepper Menthol Blue Cheese Chemical
1 ( 1 ) " " " " " " " " " " " " " " " " " " " "
2 ( 1 ) " " " " " " " " " " " " " " " " " " " "
3 ( 1 ) " " " " " " " " " " " " " " " " " " " "
4 ( 1 ) " " " " " " " " " " " " " " " " " " " "
5 ( 1 ) " " " " " " " " " " " " " " " " " " " "
6 ( 1 ) " " " " " " " " " " " " " " " " " " " "
7 ( 1 ) " " " " " " " " " " " " " " " " " " " "
8 ( 1 ) " " " " " " " " " " " " " " " " " " " "
9 ( 1 ) " " " " " " " " " " " " " " " " " " " "
10 ( 1 ) " " " " " " " " " " " " " " " " " " " "
11 ( 1 ) " " " " " " " " " " " " " " " " " " " "
12 ( 1 ) " " " " " " " " " " " " " " " " " " " "
13 ( 1 ) " " " " " " " " " " " " " " " " " " " "
14 ( 1 ) " " " " " " " " " " " " " " " " " " " "
15 ( 1 ) " " " " " " " " "*" " " " " " " " " " "
16 ( 1 ) " " " " " " " " "*" " " " " " " " " " "
17 ( 1 ) " " " " " " " " "*" " " " " " " " " " "
18 ( 1 ) " " " " " " " " "*" " " " " " " " " " "
19 ( 1 ) " " " " " " " " "*" " " "*" " " " " " "
20 ( 1 ) " " " " " " " " "*" " " "*" " " " " " "
21 ( 1 ) " " " " "*" " " "*" " " "*" " " " " " "
22 ( 1 ) " " " " "*" " " "*" " " "*" " " " " " "
23 ( 1 ) " " " " "*" " " "*" " " "*" " " " " " "
24 ( 1 ) "*" " " "*" " " "*" " " "*" " " " " " "
25 ( 1 ) "*" " " "*" " " "*" " " "*" " " " " " "
26 ( 1 ) "*" " " "*" " " "*" " " "*" " " " " " "
27 ( 1 ) "*" " " "*" " " "*" " " "*" " " " " " "
28 ( 1 ) "*" "*" "*" " " "*" " " "*" " " " " " "
29 ( 1 ) "*" "*" "*" "*" "*" " " "*" " " " " " "
30 ( 1 ) "*" "*" "*" "*" "*" " " "*" " " " " " "
31 ( 1 ) "*" "*" "*" "*" "*" " " "*" " " " " " "
32 ( 1 ) "*" "*" "*" "*" "*" "*" "*" " " " " " "
33 ( 1 ) "*" "*" "*" "*" "*" "*" "*" " " " " " "
34 ( 1 ) "*" "*" "*" "*" "*" "*" "*" " " " " " "
35 ( 1 ) "*" "*" "*" "*" "*" "*" "*" " " " " " "
36 ( 1 ) "*" "*" "*" "*" "*" "*" "*" " " " " " "
37 ( 1 ) "*" "*" "*" "*" "*" "*" "*" " " " " " "
38 ( 1 ) "*" "*" "*" "*" "*" "*" "*" " " " " " "
39 ( 1 ) "*" "*" "*" "*" "*" "*" "*" " " " " " "
40 ( 1 ) "*" "*" "*" "*" "*" "*" "*" " " " " "*"
41 ( 1 ) "*" "*" "*" "*" "*" "*" "*" " " " " "*"
42 ( 1 ) "*" "*" "*" "*" "*" "*" "*" " " " " "*"
43 ( 1 ) "*" "*" "*" "*" "*" "*" "*" " " " " "*"
44 ( 1 ) "*" "*" "*" "*" "*" "*" "*" " " " " "*"
45 ( 1 ) "*" "*" "*" "*" "*" "*" "*" " " " " "*"
46 ( 1 ) "*" "*" "*" "*" "*" "*" "*" " " " " "*"
47 ( 1 ) "*" "*" "*" "*" "*" "*" "*" " " " " "*"
48 ( 1 ) "*" "*" "*" "*" "*" "*" "*" " " " " "*"
49 ( 1 ) "*" "*" "*" "*" "*" "*" "*" " " " " "*"
50 ( 1 ) "*" "*" "*" "*" "*" "*" "*" " " " " "*"
51 ( 1 ) "*" "*" "*" "*" "*" "*" "*" " " " " "*"
52 ( 1 ) "*" "*" "*" "*" "*" "*" "*" " " " " "*"
53 ( 1 ) "*" "*" "*" "*" "*" "*" "*" " " " " "*"
54 ( 1 ) "*" "*" "*" "*" "*" "*" "*" " " " " "*"
55 ( 1 ) "*" "*" "*" "*" "*" "*" "*" " " " " "*"
56 ( 1 ) "*" "*" "*" "*" "*" "*" "*" " " " " "*"
57 ( 1 ) "*" "*" "*" "*" "*" "*" "*" " " "*" "*"
58 ( 1 ) "*" "*" "*" "*" "*" "*" "*" "*" "*" "*"
59 ( 1 ) "*" "*" "*" "*" "*" "*" "*" "*" "*" "*"
60 ( 1 ) "*" "*" "*" "*" "*" "*" "*" "*" "*" "*"
61 ( 1 ) "*" "*" "*" "*" "*" "*" "*" "*" "*" "*"
Mango Lemon Peach Vanilla Nutty Chestnut Tea Tobacco Tropical
1 ( 1 ) " " " " " " " " " " " " " " " " " "
2 ( 1 ) " " " " " " " " " " " " " " " " " "
3 ( 1 ) " " " " " " " " " " " " " " " " " "
4 ( 1 ) " " " " " " " " " " " " " " " " " "
5 ( 1 ) " " " " " " " " " " " " " " " " " "
6 ( 1 ) " " " " " " " " " " " " " " " " " "
7 ( 1 ) " " " " " " " " " " " " " " " " " "
8 ( 1 ) " " " " " " " " " " " " " " " " " "
9 ( 1 ) " " " " " " " " " " " " " " " " " "
10 ( 1 ) " " " " " " " " " " " " " " " " " "
11 ( 1 ) " " " " " " " " " " " " " " " " " "
12 ( 1 ) " " " " " " " " " " " " " " " " " "
13 ( 1 ) " " " " " " " " " " " " " " " " " "
14 ( 1 ) " " " " " " " " " " " " " " " " " "
15 ( 1 ) " " " " " " " " " " " " " " " " " "
16 ( 1 ) " " " " " " " " " " " " " " " " " "
17 ( 1 ) " " " " " " " " " " " " " " " " " "
18 ( 1 ) " " " " " " " " " " " " " " " " " "
19 ( 1 ) " " " " " " " " " " " " " " " " " "
20 ( 1 ) " " " " " " " " " " " " " " " " " "
21 ( 1 ) " " " " " " " " " " " " " " " " " "
22 ( 1 ) " " " " " " " " " " " " " " " " " "
23 ( 1 ) " " " " " " " " " " " " " " " " " "
24 ( 1 ) " " " " " " " " " " " " " " " " " "
25 ( 1 ) " " " " " " " " " " " " " " " " "*"
26 ( 1 ) " " " " " " " " " " " " " " " " "*"
27 ( 1 ) " " "*" " " " " " " " " " " " " "*"
28 ( 1 ) " " "*" " " " " " " " " " " " " "*"
29 ( 1 ) " " "*" " " " " " " " " " " " " "*"
30 ( 1 ) " " "*" " " " " " " " " " " " " "*"
31 ( 1 ) " " "*" "*" " " " " " " " " " " "*"
32 ( 1 ) " " "*" "*" " " " " " " " " " " "*"
33 ( 1 ) " " "*" "*" " " " " " " " " " " "*"
34 ( 1 ) " " "*" "*" " " " " " " " " " " "*"
35 ( 1 ) " " "*" "*" " " " " " " " " " " "*"
36 ( 1 ) " " "*" "*" " " " " " " " " " " "*"
37 ( 1 ) " " "*" "*" " " " " " " " " " " "*"
38 ( 1 ) " " "*" "*" " " " " "*" " " " " "*"
39 ( 1 ) " " "*" "*" " " " " "*" " " " " "*"
40 ( 1 ) " " "*" "*" " " " " "*" " " " " "*"
41 ( 1 ) " " "*" "*" " " " " "*" " " " " "*"
42 ( 1 ) "*" "*" "*" " " " " "*" " " " " "*"
43 ( 1 ) "*" "*" "*" " " " " "*" " " " " "*"
44 ( 1 ) "*" "*" "*" " " " " "*" " " " " "*"
45 ( 1 ) "*" "*" "*" "*" " " "*" " " " " "*"
46 ( 1 ) "*" "*" "*" "*" " " "*" " " " " "*"
47 ( 1 ) "*" "*" "*" "*" " " "*" " " " " "*"
48 ( 1 ) "*" "*" "*" "*" " " "*" " " " " "*"
49 ( 1 ) "*" "*" "*" "*" " " "*" " " " " "*"
50 ( 1 ) "*" "*" "*" "*" " " "*" " " " " "*"
51 ( 1 ) "*" "*" "*" "*" " " "*" " " " " "*"
52 ( 1 ) "*" "*" "*" "*" " " "*" " " " " "*"
53 ( 1 ) "*" "*" "*" "*" "*" "*" " " " " "*"
54 ( 1 ) "*" "*" "*" "*" "*" "*" " " " " "*"
55 ( 1 ) "*" "*" "*" "*" "*" "*" " " " " "*"
56 ( 1 ) "*" "*" "*" "*" "*" "*" "*" " " "*"
57 ( 1 ) "*" "*" "*" "*" "*" "*" "*" " " "*"
58 ( 1 ) "*" "*" "*" "*" "*" "*" "*" " " "*"
59 ( 1 ) "*" "*" "*" "*" "*" "*" "*" " " "*"
60 ( 1 ) "*" "*" "*" "*" "*" "*" "*" " " "*"
61 ( 1 ) "*" "*" "*" "*" "*" "*" "*" "*" "*"
Strawberry Blueberry Mint Apple Honey Lavender Lime Coffee Ammonia
1 ( 1 ) " " " " " " " " " " " " " " " " " "
2 ( 1 ) " " " " " " " " " " " " " " " " " "
3 ( 1 ) " " " " " " " " " " " " " " " " " "
4 ( 1 ) " " " " " " " " " " " " " " " " " "
5 ( 1 ) " " " " " " " " " " " " " " " " " "
6 ( 1 ) " " " " " " " " " " " " " " " " " "
7 ( 1 ) " " " " " " " " " " " " " " " " " "
8 ( 1 ) " " " " " " " " " " " " " " " " " "
9 ( 1 ) " " " " " " " " " " " " " " " " " "
10 ( 1 ) " " " " " " " " " " " " " " " " " "
11 ( 1 ) " " " " " " " " " " " " " " " " " "
12 ( 1 ) " " " " " " " " " " " " " " " " " "
13 ( 1 ) " " " " " " " " " " " " " " " " " "
14 ( 1 ) " " " " " " " " " " " " " " " " " "
15 ( 1 ) " " " " " " " " " " " " " " " " " "
16 ( 1 ) " " " " " " " " " " " " " " " " " "
17 ( 1 ) " " " " " " " " " " " " " " " " " "
18 ( 1 ) " " " " " " " " " " " " " " " " " "
19 ( 1 ) " " " " " " " " " " " " " " " " " "
20 ( 1 ) " " " " " " " " " " " " "*" " " " "
21 ( 1 ) " " " " " " " " " " " " "*" " " " "
22 ( 1 ) " " " " " " " " " " " " "*" " " " "
23 ( 1 ) " " " " " " " " " " " " "*" "*" " "
24 ( 1 ) " " " " " " " " " " " " "*" "*" " "
25 ( 1 ) " " " " " " " " " " " " "*" "*" " "
26 ( 1 ) " " " " " " " " " " " " "*" "*" " "
27 ( 1 ) " " " " " " " " " " " " "*" "*" " "
28 ( 1 ) " " " " " " " " " " " " "*" "*" " "
29 ( 1 ) " " " " " " " " " " " " "*" "*" " "
30 ( 1 ) " " " " " " " " " " " " "*" "*" " "
31 ( 1 ) " " " " " " " " " " " " "*" "*" " "
32 ( 1 ) " " " " " " " " " " " " "*" "*" " "
33 ( 1 ) " " "*" " " " " " " " " "*" "*" " "
34 ( 1 ) " " "*" " " " " "*" " " "*" "*" " "
35 ( 1 ) " " "*" " " " " "*" " " "*" "*" " "
36 ( 1 ) " " "*" " " " " "*" " " "*" "*" "*"
37 ( 1 ) " " "*" " " " " "*" " " "*" "*" "*"
38 ( 1 ) " " "*" " " " " "*" " " "*" "*" "*"
39 ( 1 ) " " "*" " " " " "*" "*" "*" "*" "*"
40 ( 1 ) " " "*" " " " " "*" "*" "*" "*" "*"
41 ( 1 ) " " "*" " " " " "*" "*" "*" "*" "*"
42 ( 1 ) " " "*" " " " " "*" "*" "*" "*" "*"
43 ( 1 ) "*" "*" " " " " "*" "*" "*" "*" "*"
44 ( 1 ) "*" "*" " " "*" "*" "*" "*" "*" "*"
45 ( 1 ) "*" "*" " " "*" "*" "*" "*" "*" "*"
46 ( 1 ) "*" "*" " " "*" "*" "*" "*" "*" "*"
47 ( 1 ) "*" "*" " " "*" "*" "*" "*" "*" "*"
48 ( 1 ) "*" "*" " " "*" "*" "*" "*" "*" "*"
49 ( 1 ) "*" "*" "*" "*" "*" "*" "*" "*" "*"
50 ( 1 ) "*" "*" "*" "*" "*" "*" "*" "*" "*"
51 ( 1 ) "*" "*" "*" "*" "*" "*" "*" "*" "*"
52 ( 1 ) "*" "*" "*" "*" "*" "*" "*" "*" "*"
53 ( 1 ) "*" "*" "*" "*" "*" "*" "*" "*" "*"
54 ( 1 ) "*" "*" "*" "*" "*" "*" "*" "*" "*"
55 ( 1 ) "*" "*" "*" "*" "*" "*" "*" "*" "*"
56 ( 1 ) "*" "*" "*" "*" "*" "*" "*" "*" "*"
57 ( 1 ) "*" "*" "*" "*" "*" "*" "*" "*" "*"
58 ( 1 ) "*" "*" "*" "*" "*" "*" "*" "*" "*"
59 ( 1 ) "*" "*" "*" "*" "*" "*" "*" "*" "*"
60 ( 1 ) "*" "*" "*" "*" "*" "*" "*" "*" "*"
61 ( 1 ) "*" "*" "*" "*" "*" "*" "*" "*" "*"
Minty Tree Fruit Butter Pineapple Tar Rose Plum Pear
1 ( 1 ) " " " " " " " " " " " " " " " " " "
2 ( 1 ) " " " " " " " " " " " " " " " " " "
3 ( 1 ) " " " " " " " " " " " " " " " " " "
4 ( 1 ) " " " " " " " " " " " " " " " " " "
5 ( 1 ) " " " " " " " " " " " " " " " " " "
6 ( 1 ) " " " " " " " " " " " " " " " " " "
7 ( 1 ) " " " " " " " " " " " " " " " " " "
8 ( 1 ) " " " " " " " " " " " " " " " " " "
9 ( 1 ) " " " " " " " " " " " " " " " " " "
10 ( 1 ) " " " " " " " " " " " " " " " " " "
11 ( 1 ) " " " " " " " " " " " " " " " " " "
12 ( 1 ) " " " " " " " " " " " " " " " " " "
13 ( 1 ) " " " " " " " " " " " " " " " " " "
14 ( 1 ) "*" " " " " " " " " " " " " " " " "
15 ( 1 ) "*" " " " " " " " " " " " " " " " "
16 ( 1 ) "*" " " " " " " " " " " " " " " " "
17 ( 1 ) "*" " " " " " " " " " " " " " " " "
18 ( 1 ) "*" " " " " " " " " " " " " " " " "
19 ( 1 ) "*" " " " " " " " " " " " " " " " "
20 ( 1 ) "*" " " " " " " " " " " " " " " " "
21 ( 1 ) "*" " " " " " " " " " " " " " " " "
22 ( 1 ) "*" " " " " " " " " " " " " " " " "
23 ( 1 ) "*" " " " " " " " " " " " " " " " "
24 ( 1 ) "*" " " " " " " " " " " " " " " " "
25 ( 1 ) "*" " " " " " " " " " " " " " " " "
26 ( 1 ) "*" " " " " " " " " " " " " " " " "
27 ( 1 ) "*" " " " " " " " " " " " " " " " "
28 ( 1 ) "*" " " " " " " " " " " " " " " " "
29 ( 1 ) "*" " " " " " " " " " " " " " " " "
30 ( 1 ) "*" " " " " " " " " " " " " " " " "
31 ( 1 ) "*" " " " " " " " " " " " " " " " "
32 ( 1 ) "*" " " " " " " " " " " " " " " " "
33 ( 1 ) "*" " " " " " " " " " " " " " " " "
34 ( 1 ) "*" " " " " " " " " " " " " " " " "
35 ( 1 ) "*" " " " " " " " " " " " " " " " "
36 ( 1 ) "*" " " " " " " " " " " " " " " " "
37 ( 1 ) "*" " " " " " " " " " " " " " " " "
38 ( 1 ) "*" " " " " " " " " " " " " " " " "
39 ( 1 ) "*" " " " " " " " " " " " " " " " "
40 ( 1 ) "*" " " " " " " " " " " " " " " " "
41 ( 1 ) "*" " " " " " " " " " " " " " " " "
42 ( 1 ) "*" " " " " " " " " " " " " " " " "
43 ( 1 ) "*" " " " " " " " " " " " " " " " "
44 ( 1 ) "*" " " " " " " " " " " " " " " " "
45 ( 1 ) "*" " " " " " " " " " " " " " " " "
46 ( 1 ) "*" "*" " " " " " " " " " " " " " "
47 ( 1 ) "*" "*" " " " " " " "*" " " " " " "
48 ( 1 ) "*" "*" " " " " " " "*" " " " " " "
49 ( 1 ) "*" "*" " " " " " " "*" " " " " " "
50 ( 1 ) "*" "*" " " "*" " " "*" " " " " " "
51 ( 1 ) "*" "*" " " "*" " " "*" "*" " " " "
52 ( 1 ) "*" "*" " " "*" " " "*" "*" " " "*"
53 ( 1 ) "*" "*" " " "*" " " "*" "*" " " "*"
54 ( 1 ) "*" "*" " " "*" " " "*" "*" " " "*"
55 ( 1 ) "*" "*" " " "*" " " "*" "*" " " "*"
56 ( 1 ) "*" "*" " " "*" " " "*" "*" " " "*"
57 ( 1 ) "*" "*" " " "*" " " "*" "*" " " "*"
58 ( 1 ) "*" "*" " " "*" " " "*" "*" " " "*"
59 ( 1 ) "*" "*" " " "*" "*" "*" "*" " " "*"
60 ( 1 ) "*" "*" " " "*" "*" "*" "*" "*" "*"
61 ( 1 ) "*" "*" " " "*" "*" "*" "*" "*" "*"
leaps BIC
Model 13 -1159.2405
Model 14 -1155.1099
Model 15 -1150.6525
Model 16 -1144.7077
Model 17 -1138.9550
Model 18 -1133.3869
Model 19 -1127.5767
Model 20 -1121.7124
Model 21 -1115.7266
Model 22 -1109.1888
Model 23 -1102.7709
Model 24 -1096.3666
Model 25 -1089.9863
Model 26 -1083.7533
Model 27 -1077.4824
Model 28 -1071.4119
Model 29 -1065.5553
Model 30 -1060.1198
Model 12 -1057.7093
Model 31 -1054.8088
Model 32 -1049.5528
Model 33 -1043.8408
Model 34 -1037.9501
Model 35 -1032.0594
Model 36 -1026.2087
Model 37 -1020.2696
Model 38 -1014.3434
Model 39 -1008.1102
Model 40 -1001.7643
Model 41 -994.9492
Model 42 -988.3507
Model 43 -981.6881
Model 44 -974.8097
Model 45 -967.6551
Model 46 -960.5650
Model 11 -955.2681
Model 47 -953.3796
Model 48 -946.0997
Model 49 -938.7930
Model 50 -931.4421
Model 51 -924.0578
Model 52 -916.6737
Model 53 -909.2966
Model 54 -901.7930
Model 55 -894.2908
Model 56 -886.7031
Model 57 -879.0910
Model 58 -871.4425
Model 10 -869.8003
Model 59 -863.7226
Model 60 -855.9846
Model 61 -848.2431
Model 9 -789.0044
Model 8 -696.0323
Model 7 -614.3485
Model 6 -552.6699
Model 5 -494.0750
Model 4 -418.6211
Model 3 -348.0542
Model 2 -279.0113
Model 1 -186.0198
leapsleaps Creative Energetic Tingly Euphoric Relaxed
"*" "*" "*" "*" "*"
Aroused Happy Uplifted Hungry Talkative
"*" "*" "*" "*" "*"
Giggly Focused Sleepy Earthy Sweet
"*" "*" "*" " " " "
Citrus Flowery Violet Diesel `Spicy/Herbal`
" " " " " " " " " "
Sage Woody Apricot Grapefruit Orange
" " " " " " " " " "
Pungent Grape Pine Skunk Berry
" " " " " " " " " "
Pepper Menthol Blue Cheese Chemical
" " " " " " " " " "
Mango Lemon Peach Vanilla Nutty
" " " " " " " " " "
Chestnut Tea Tobacco Tropical Strawberry
" " " " " " " " " "
Blueberry Mint Apple Honey Lavender
" " " " " " " " " "
Lime Coffee Ammonia Minty Tree
" " " " " " " " " "
Fruit Butter Pineapple Tar Rose
" " " " " " " " " "
Plum Pear
" " " "
Start with one variable that you think is best.
Add the next best variable.
Test whether it was worth adding.
Keep going until it’s not worth adding any more variables.
Open Activity-Variable-Selection
Determine the best model via backwards selection.
Fit that model to the data and report results.
Determine the best model via forwards selection.
Fit that model to the data and report results.